Information providing device, information providing method, and storage medium

ABSTRACT

An information providing device of an embodiment is an information providing device including an acquirer that acquires personal information of a plurality of occupants in a moving body, a processor that generates integrated personal information by combining at least a part of the personal information of the plurality of occupants acquired by the acquirer, and an information provider that provides information to the plurality of occupants based on the integrated personal information processed by the processor.

Priority is claimed on Japanese Patent Application No. 2019-060022,filed Mar. 27, 2019, the content of which is incorporated herein byreference.

BACKGROUND Field of the Invention

The present invention relates to an information providing device, aninformation providing method, and a storage medium.

Description of Related Art

In the related art, there has been disclosed a technology related to anagent function of providing information on driving assistance, controlof a vehicle, other applications and the like according to a request ofan occupant of the vehicle while interacting with the occupant (forexample, see Japanese Unexamined Patent Application, First PublicationNo. 2006-335231). Furthermore, Patent Document 1 discloses a technologyin which, when there are a plurality of passengers, an agent characterdisplayed on a display means in a driver seat is also displayed on adisplay means of a passenger seat or a rear seat.

SUMMARY

However, in the aforementioned technology, when responding to theplurality of passengers in the vehicle, no consideration is given towhich passenger is to be responded to. Therefore, it may not be possibleto send an appropriate response to a user.

The present invention is achieved in view of the problems describedabove, and one object of the present invention is to provide aninformation providing device, an information providing method, and astorage medium, by which it is possible to provide more appropriateinformation to an occupant of a moving body.

Solution to Problem

An information providing device, an information providing method, and astorage medium according to the invention employ the followingconfigurations.

(1): An information providing device according to an aspect of theinvention is an information providing device including an acquirer thatacquires personal information of a plurality of occupants in a movingbody, a processor that generates integrated personal information bycombining at least a part of the personal information of the pluralityof occupants acquired by the acquirer, and an information provider thatprovides information to the plurality of occupants based on theintegrated personal information processed by the processor.

(2) In the aspect of the aforementioned (1), the information providerfurther includes a display controller that controls a display to displayan agent image that communicates with the plurality of occupants, and anagent functional unit that provides a service including allowing anoutput to output a sound response according to sounds of the pluralityof occupants, wherein the processor combines at least a part of thepersonal information of the plurality of occupants based on an area ofthe display where the agent image is displayed.

(3) In the aspect of the aforementioned (2), the processor changes thedegree of influence of the personal information of the occupantsincluded in the integrated personal information, based on a displayposition of the agent image and respective positions of the plurality ofoccupants in the moving body.

(4) In the aspect of the aforementioned (3), the processor increases thedegree of influence of the personal information of the occupantsincluded in the integrated personal information with respect to anoccupant located at a position near the display position of the agentimage among the plurality of occupants.

(5) In the aspect of the aforementioned (3), the processor allows thedegree of influence of personal information of an occupant, who drivesthe moving body, among the personal information included in theintegrated personal information to be larger than the degree ofinfluence of personal information of other occupants.

(6) In the aspect of the aforementioned (3), the processor changes theratio of the personal information of the occupants included in theintegrated personal information, based on the display position of theagent image and respective positions of the plurality of occupants inthe moving body.

(7) In the aspect of the aforementioned (2), the display has a displayarea large enough for two or more of the plurality of occupants torecognize that the agent image is displayed at a position facing the twoor more occupants.

(8) An information providing method according to another aspect of thepresent invention is an information providing method includingacquiring, by an information providing device, personal information of aplurality of occupants in a moving body, generating, by the informationproviding device, integrated personal information by combining at leasta part of the acquired personal information of the plurality ofoccupants, and providing, by the information providing device,information to the plurality of occupants based on the generatedintegrated personal information.

(9) A storage medium according to another aspect of the presentinvention is a computer readable non-transitory storing medium storing aprogram, which causes an information providing device to performacquiring personal information of a plurality of occupants in a movingbody, generating integrated personal information by combining at least apart of the acquired personal information of the plurality of occupants,and providing information to the plurality of occupants based on thegenerated integrated personal information.

According to the aspects of (1) to (9), it is possible to provide moreappropriate information to an occupant of a moving body.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration diagram of an agent system including an agentdevice.

FIG. 2 is a diagram illustrating a configuration of the agent deviceaccording to a first embodiment and devices mounted on a vehicle.

FIG. 3 is a diagram illustrating an arrangement example of a display andoperation device.

FIG. 4 is a diagram illustrating an arrangement example of a speakerunit.

FIG. 5 is a diagram illustrating a configuration of an agent serveraccording to the first embodiment and a part of the configuration of theagent device.

FIG. 6 is a diagram illustrating an example of the content of a personalprofile.

FIG. 7 is a diagram illustrating an example of the content of anintegrated profile.

FIG. 8 is a diagram for explaining weights set by an integrated profilegenerator.

FIG. 9 is a diagram for explaining response content when an agent imageis displayed at a position near an occupant U1.

FIG. 10 is a diagram for explaining response content when an agent imageis displayed at a position near an occupant U2.

FIG. 11 is a flowchart illustrating an example of the flow of anintegrated profile generation process performed by the agent device ofthe first embodiment.

FIG. 12 is a sequence diagram illustrating an example of the flow of aninformation providing process performed by the agent system of the firstembodiment.

FIG. 13 is a diagram illustrating a configuration of an agent deviceaccording to a second embodiment and devices mounted on a vehicle.

FIG. 14 is a diagram illustrating a configuration of an agent serveraccording to the second embodiment and a part of the configuration ofthe agent device.

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of an information providing device, aninformation providing method, and a storage medium of the presentinvention will be described with reference to the drawings.

First Embodiment

In the first embodiment, an agent device is used as an example of aninformation providing device. The agent device is a device thatimplements a part or all of an agent system. Hereinafter, as an exampleof the agent device, an agent device mounted on a vehicle (hereinafter,a vehicle M), which is an example of a moving body, and having aplurality of types of agent functions will be described. In applying thepresent invention, the agent device does not always have a plurality oftypes of agent functions, and may be a portable terminal device such asa smart phone; however, in the following description, an agent devicemounted on a vehicle and having a plurality of types of agent functionsis assumed. The agent function refers to, for example, a function ofproviding various types of information or controlling various devicesbased on requests (commands) included in utterance of occupants (anexample of users) of the vehicle M or mediating a network service whileinteracting with the occupants. A plurality of types of agents maydiffer in functions to be performed, processing procedures, control, andoutput modes and contents. Some of the agent functions may have afunction of performing control and the like of devices in the vehicle(for example, devices related to driving control and vehicle bodycontrol).

The agent functions are implemented integrally using a natural languageprocessing function (function of understanding the structure and meaningof a text), a conversation management function, a network searchfunction of searching for other devices via a network or searching for apredetermined database held by own device, and the like, in addition toa sound recognition function of recognizing sounds of occupants(function of converting sounds to texts). Some or all of these functionsmay be implemented by an artificial intelligence (AI) technology. A partof a configuration for performing these functions (particularly, a soundrecognition function and a natural language processing andinterpretation function) may be mounted on an agent server (externaldevice) capable of communicating with an in-vehicle communication deviceof the vehicle M or a general purpose communication device brought intothe vehicle M. In the following description, it is assumed that a partof the configuration is mounted on the agent server and the agent deviceand the agent server implement an agent system in cooperation with eachother. A service providing entity (service entity) that virtuallyappears in cooperation of the agent device and the agent server isreferred to as an agent.

<Overall Configuration>

FIG. 1 is a configuration diagram of an agent system 1 including anagent device 100. The agent system 1 includes, for example, the agentdevice 100, a plurality of agent servers 200-1 to 200-3, . . . . It isassumed that the number following the hyphen at the end of the referencenumerals are identifiers for distinguishing agents. When it is notnecessary to distinguish between the agent servers, they may be simplyreferred to as the agent server 200. FIG. 1 illustrates three agentservers 200, but the number of agent servers 200 may be two, or four ormore. The agent servers 200 are operated by providers of different agentsystems. Consequently, the agents in the present embodiment are agentsimplemented by different providers. The provider includes, for example,an automobile manufacturer, a network service provider, an electroniccommerce provider, a seller or a manufacturer of a portable terminal,and the like, and an arbitrary entity (corporation, organization,individual and the like) may be a provider of the agent system.

The agent device 100 communicates with the agent server 200 via anetwork NW. The network NW includes, for example, some or all of theInternet, a cellular network, a Wi-Fi network, a wide area network(WAN), a local area network (LAN), a public line, a telephone line, aradio base station, and the like. Various web servers 300 are connectedto the network NW, and the agent server 200 or the agent device 100 canacquire web pages from the various web servers 300 via the network NW.

The agent device 100 interacts with the occupants of the vehicle M,transmits sounds of the occupants to the agent server 200, and presentsa response obtained from the agent server 200 to the occupants in theform of sound output or image display.

[Vehicle]

FIG. 2 is a diagram illustrating the configuration of the agent device100 according to the first embodiment and devices mounted on the vehicleM. In the vehicle M, for example, one or more microphones 10, a displayand operation device 20, a speaker unit 30, a navigation device 40, avehicle device 50, an in-vehicle communication device 60, an occupantrecognition device 80, and the agent device 100 are mounted. There is acase where a general purpose communication device 70 such as a smartphone is brought into a vehicle interior and is used as a communicationdevice. These devices are mutually connected by a multiplexcommunication line such as a controller area network (CAN) communicationline, a serial communication line, a wireless communication network, andthe like. The configuration illustrated in FIG. 2 is merely an example,a part of the configuration may be omitted, or another configuration maybe further added. At least one of the display and operation device 20and the speaker unit 30 is an example of an “output”.

The microphone 10 is a sound collector that collects sounds generatedinside the vehicle interior. The display and operation device 20 is adevice (for example, a device group) capable of displaying an image andreceiving an input operation. The display and operation device 20includes, for example, a display device configured as a touch panel. Thedisplay and operation device 20 may further include a head up display(HUD) or a mechanical input device. The speaker unit 30 includes, forexample, a plurality of speakers (sound outputs) arranged at differentpositions in the vehicle interior. The display and operation device 20may be shared by the agent device 100 and the navigation device 40.Details thereof will be described below.

The navigation device 40 includes a navigation human machine interface(HMI), a location positioning device such as a global positioning system(GPS), a storage device that stores map information, and a controldevice (navigation controller) that performs route search, and the like.Some or all of the microphone 10, the display and operation device 20,and the speaker unit 30 may be used as a navigation HMI. The navigationdevice 40 searches for a route (navigation route) for moving to adestination input by an occupant from the position of the vehicle Mspecified by the location positioning device, and outputs guidanceinformation by using the navigation HMI such that the vehicle M cantravel along the route. The route search function may be provided in anavigation server accessible via the network NW. In such a case, thenavigation device 40 acquires the route from the navigation server andoutputs the guidance information. The agent device 100 may beconstructed based on the navigation controller, and in such a case, thenavigation controller and the agent device 100 are integrally configuredon hardware.

The vehicle device 50 includes, for example, a driving force outputdevice such as an engine and a travel motor, an engine starting motor, adoor lock device, a door opening and closing device, a window, anopening and closing device of the window, an opening and closing controldevice of the window, a seat, a seat position control device, a rearviewmirror and its angle position control device, a lighting device insideand outside of the vehicle and its control device, a wiper and defoggerand its control device, a direction indicator and its control device, anair conditioner, a vehicle information device of information on a traveldistance and tire air pressure, information on remaining fuel, and thelike.

The in-vehicle communication device 60 is, for example, a wirelesscommunication device capable of accessing the network NW by using acellular network or a Wi-Fi network.

The occupant recognition device 80 includes, for example, a seatingsensor, a vehicle interior camera, an image recognition device, a soundrecognition device, and the like. The seating sensor includes a pressuresensor provided below the seat, a tension sensor attached to a seatbelt, and the like. The vehicle interior camera is a charge coupleddevice (CCD) camera or a complementary metal oxide semiconductor (CMOS)camera provided in the vehicle interior. The image recognition deviceanalyzes an image of the vehicle interior camera, and recognizesinformation on the presence or absence of an occupant for each seat,orientation of a face, feature information of the face, and the like.The feature information of the face is, for example, information on theskin color, contour shape, arrangement of characteristic parts (eyes,nose, and mouth), and shape of the face. The sound recognition devicerecognizes sound feature information of each occupant from soundsacquired from the microphone 10 and the like. The sound featureinformation is, for example, information on feature amounts based onsound pitch, intonation, rhythm (sound pitch pattern), features of one'sway speaking such as pose, Mel frequency Cepstrum coefficients, and thelike. Based on the intensity of the sounds collected by the plurality ofmicrophones 10, a time difference when the same sound has beencollected, and the like, the occupant recognition device 80 may estimatepositions, where the sounds have been output, and recognize the seatingpositions of the occupants from the estimated positions. The occupantrecognition device 80 may recognize a speaker from the movement of amouth obtained from the image and the positions where the sounds havebeen output. The occupant recognition device 80 may recognize theposition on vehicle interior space coordinates where each occupant isseated, in correlation with the aforementioned face and sound featureinformation. That is, in the present embodiment, the occupantrecognition device 80 is an example of a seating position recognizer.

FIG. 3 is a diagram illustrating an arrangement example of the displayand operation device 20. The display and operation device 20 includes,for example, a first display 22, a second display 24, and an operationswitch ASSY 26. The display and operation device 20 may further includea HUD 28. The display and operation device 20 may further include ameter display 29 provided on a portion of an installment panel facing adriver seat DS. A combination of some or all of the first display 22,the second display 24, the HUD 28, and the meter display 29 is anexample of a “display”.

The vehicle M includes, for example, the driver seat DS provided with asteering wheel SW and an assistant seat AS provided in a vehicle widthdirection (Y direction in the drawing) with respect to the driver seatDS. The first display 22 has a wide display area enough for two or moreof a plurality of occupants to recognize that an agent image, which willbe described below, is displayed at a position facing the occupants.Specifically, the first display 22 is a horizontally long display deviceextending from the vicinity of the left side of the driver seat DS inthe installment panel to a position facing the left end portion of theassistant seat AS. When the meter display 29 is not provided on thevehicle M, the first display 22 may be extended to the vicinity of theright end portion of the driver seat DS, and content, which is to bedisplayed on the meter display 29 to be described below, may bedisplayed on an area on the right side (driver seat DS side) when viewedfrom the front of the first display 22.

The second display 24 is installed at an intermediate position betweenthe driver seat DS and the assistant seat AS in the vehicle widthdirection, and below the first display. For example, each of the firstdisplay 22 and the second display 24 is configured as a touch panel andincludes a liquid crystal display (LCD), an organic electroluminescence(EL), a plasma display, and the like as a display. The operation switchASSY 26 is formed by integrating dial switches, button switches, and thelike. The HUD 28 is, for example, a device for visually recognizing animage by superimposing the image on a landscape, and as an example, theHUD 28 projects light including an image on a front windshield or acombiner of the vehicle M, thereby allowing the occupants to visuallyrecognize a virtual image. The meter display 29 is, for example, an LCD,an organic EL, and the like, and displays an instrument such as aspeedometer and a tachometer. The display and operation device 20outputs the content of operations performed by the occupants to theagent device 100. The content displayed by the aforementioned eachdisplay may be determined by the agent device 100.

FIG. 4 is a diagram illustrating an arrangement example of the speakerunit 30. The speaker unit 30 includes, for example, speakers 30A to 30H.The speaker 30A is installed on a window post (so-called a pillar A) onthe driver seat DS side. The speaker 30B is installed below a door nearthe driver seat DS. The speaker 30C is installed on a window post on theassistant seat AS side. The speaker 30D is installed below a door nearthe assistant seat AS. The speaker 30E is installed below a door near aright rear seat BS1 side. The speaker 30F is installed below a door neara left rear seat BS2 side. The speaker 30G is installed in the vicinityof the second display 24. The speaker 30H is installed on a ceiling(roof) of the vehicle interior.

In such an arrangement, for example, when sounds are exclusively outputto the speakers 30A and 30B, a sound image is localized near the driverseat DS. The “localizing the sound image” means, for example,determining a spatial position of a sound source perceived by anoccupant by adjusting the loudness of the sounds transferred to theright and left ears of the occupant. When sounds are exclusively outputto the speakers 30C and 30D, a sound image is localized near theassistant seat AS. When sounds are exclusively output to the speaker30E, a sound image is localized near the front of the vehicle interior,and when sounds are exclusively output to the speaker 30F, a sound imageis localized near the upper portion of the vehicle interior. When soundsare exclusively output to the speaker 30G, a sound image is localizednear the front of the vehicle interior, and when sounds are exclusivelyoutput to the speaker 30H, a sound image is localized near the upperportion of the vehicle interior. The present invention is not limitedthereto, and the speaker unit 30 can localize a sound image at anarbitrary position in the vehicle interior by adjusting the distributionof sounds output from each speaker by using a mixer or an amplifier.

[Agent Device]

Returning to FIG. 2, the agent device 100 includes a manager 110, agentfunctions 150-1 to 150-3, a pairing application executor 160, and astorage 170. The manager 110 includes, for example, an acousticprocessor 112, a wake-up (WU) judger 114 for each agent, and an outputcontroller 120. Hereinafter, when no distinction is made among the agentfunctions, they are simply referred to as the agent function 150. Theillustration of the three agent functions 150 is merely an examplecorresponding to the number of agent servers 200 in FIG. 1, and thenumber of agent functions 150 may be two, or four or more. The softwarearrangement illustrated in FIG. 2 is simply illustrated for the purposeof description, and can be actually modified arbitrarily such that, forexample, the manager 110 may be interposed between the agent function150 and the in-vehicle communication device 60. Hereinafter, an agentthat appears in cooperation of the agent function 150-1 and the agentserver 200-1 may be referred to as an “agent 1”, an agent that appearsin cooperation of the agent function 150-2 and the agent server 200-2may be referred to as an “agent 2”, and an agent that appears incooperation of the agent function 150-3 and the agent server 200-3 maybe referred to as an “agent 3”. In the first embodiment, a combinationof some or all of the agent function 150 and the output controller 120is an example of an “information provider”.

Each component of the agent device 100 is implemented by, for example, ahardware processor such as a central processor (CPU) that executes aprogram (software). Some or all of these components may be implementedby hardware (a circuit unit: including circuitry) such as a large scaleintegration (LSI), an application specific integrated circuit (ASIC), afield-programmable gate array (FPGA), and a graphics processor (GPU), ormay be implemented by cooperation of software and hardware. The programmay be stored in advance in a storage device (storage device including anon-transitory storage medium) such as an HDD and a flash memory, or maybe installed in the HDD and the flash memory when a detachable storagemedium (non-transitory storage medium) storing the program, such as aDVD and a CD-ROM, is mounted on a drive device.

The storage 170 is implemented by the aforementioned various storagedevices. The storage 170 stores, for example, data such as an integratedprofile 172 and programs. The integrated profile 172 is data generatedby an integrated profile generator 154 to be described below. Details ofthe content of the integrated profile 172 will be described below.

The manager 110 functions by executing a program such as an operatingsystem (OS) and middleware.

The acoustic processor 112 of the manager 110 receives the soundscollected from the microphone 10, and performs an acoustic process onthe received sounds so that it is brought into a state suitable forrecognizing a wake-up word set in advance for each agent. The wake-upword includes, for example, a word, a phrase and the like for activatinga target agent. The acoustic process includes, for example, noiseremoval by filtering of a band pass filter and the like, soundamplification, and the like. The acoustic processor 112 outputs thesounds subjected to the acoustic process to the WU judger 114 for eachagent and an activated agent function.

The WU judger 114 for each agent exists in correspondence with each ofthe agent functions 150-1 to 150-3, and recognizes a wake-up word set inadvance for each agent. The WU judger 114 for each agent recognizes themeaning of the sounds from the sounds (sound stream) subjected to theacoustic process. First, the WU judger 114 for each agent detects asound section based on the amplitude and zero crossing of a soundwaveform in the sound stream. The WU judger 114 for each agent mayperform section detection based on sound identification and non-soundidentification on a frame basis based on a Gaussian mixture model (GMM).

Next, the WU judger 114 for each agent converts the sound in thedetected sound section into a text and uses the text as characterinformation. Then, the WU judger 114 for each agent determines whetherthe character information converted into text corresponds to the wake-upword. When it is determined that the character information is thewake-up word, the WU judger 114 for each agent activates a correspondingagent function 150. A function corresponding to the WU judger 114 foreach agent may be provided in the agent server 200. In such a case, themanager 110 transmits the sound stream subjected to the acoustic processby the acoustic processor 112 to the agent server 200, and when theagent server 200 determines that the sound stream is the wake-up word,the manager 110 activates the agent function 150 according to aninstruction from the agent server 200. Each agent function 150 mayalways be activated and may determine the wake-up word by itself. Insuch a case, the manager 110 needs not include the WU judger 114 foreach agent.

When the WU judger 114 for each agent recognizes a termination wordincluded in the uttered sound in a procedure similar to theaforementioned procedure and an agent corresponding to the terminationword is in an activated state (hereinafter, referred to as “activated”as needed), the WU judger 114 for each agent terminates (stops) theactivated agent function. The activation and termination of the agentmay be executed, for example, by receiving a predetermined operationfrom the display and operation device 20; however, hereinafter, anexample of sound activation and stopping by sounds will be described. Anactivated agent may be stopped when no sound input is received for apredetermined time or more.

The output controller 120 controls the display or the speaker unit 30such that information on a response result and the like is outputaccording to an instruction from the manager 110 or the agent function150, thereby providing a service and the like to the occupants. Theoutput controller 120 includes, for example, a display controller 122and a sound controller 124.

The display controller 122 controls an image to be displayed in apredetermined area of the display according to an instruction from theoutput controller 120. Hereinafter, a description will be given assumingthat the first display 22 displays an agent-related image. The displaycontroller 122 generates, for example, an image of a personified agent(hereinafter, referred to as an agent image) that communicates with theoccupants in the vehicle interior, and controls the first display 22such that the generated agent image is displayed, under the control ofthe output controller 120. The agent image is, for example, an image ina mode of talking to the occupants. The agent image may include, forexample, a face image to the extent that an expression and a facedirection are recognized by at least a viewer (occupant). For example,in the agent image, parts simulating eyes and a nose are represented ina face area and an expression and a face direction are recognized basedon the positions of the parts in the face area. The agent image isperceived three-dimensionally and includes a head image in athree-dimensional space, so that the face direction of the agent may berecognized by the viewer, or includes an image of a body (torso andlimbs), so that the operation, behavior, posture and the like of theagent may be recognized by the viewer. The agent image may be ananimation image.

The display controller 122 may, for example, control the agent image tobe displayed in a display area near the position of an occupantrecognized by the occupant recognition device 80, or perform controlsuch that an agent image in which a face is directed to the position ofthe occupant is generated and displayed. When a plurality of occupantsin the vehicle M are recognized by the occupant recognition device 80,the display controller 122 may move the agent image toward a speakerrecognized by the occupant recognition device 80, or may move the agentimage to a position visually recognizable from the positions of theplurality of occupants. The display controller 122 may move the agentimage to a position specified by an occupant's operation. The occupant'soperation includes, for example, an operation in which the occupanttouches a display position of the agent image displayed on the firstdisplay 22 with his/her finger and slides the touched finger in anydirection on a screen, an operation by a gesture of the occupantrepresenting the movement of the agent image in a non-touched state, anoperation by sounds instructing the movement of the agent image, and thelike. The movement of the agent image includes, for example, movement onthe same display (for example, movement from the right to the left onthe first display 22), movement between other displays (for example,movement from the first display 22 to the second display 24), and thelike.

The sound controller 124 controls some or all of the speakers includedin the speaker unit 30 to output sounds according to an instruction fromthe output controller 120. The sound controller 124 may use a pluralityof speaker units 30 and perform control for localizing sound images ofagent sounds at a position corresponding to the display position of theagent image. The position corresponding to the display position of theagent image is, for example, a position where an occupant is predictedto perceive that the agent image is uttering agent sounds, andspecifically, is a position near the display position of the agent image(for example, within 2 cm to 3 cm).

The agent function 150 provides a service including causing an agent toappear in cooperation with a corresponding agent server 200 and causingthe output to output a sound response according to an utterance of theoccupant of the vehicle. The agent function 150 may include an agentfunction to which authority for controlling the vehicle M (for example,the vehicle device 50) has been given. Some of the agent functions 150may communicate with the agent server 200 in cooperation with thegeneral purpose communication device 70 via the pairing applicationexecutor 160. For example, authority for controlling the vehicle M (forexample, the vehicle device 50) has been given to the agent function150-1. The agent function 150-1 communicates with the agent server 200-1via the in-vehicle communication device 60. The agent function 150-2communicates with the agent server 200-2 via the in-vehiclecommunication device 60. The agent function 150-3 communicates with theagent server 200-3 in cooperation with the general purpose communicationdevice 70 via the pairing application executor 160.

The agent function 150-1 includes a personal profile acquirer 152-1 andan integrated profile generator 154-1, the agent function 150-2 includesa personal profile acquirer 152-2 and an integrated profile generator154-2, and the agent function 150-3 includes a personal profile acquirer152-3 and an integrated profile generator 154-3. The personal profileacquirer 152 is an example of an “acquirer”. The integrated profilegenerator 154 is an example of a “processor”. The personal profileacquirer 152 transmits the occupant recognition result of the occupantrecognition device 80 to the agent server 200, and acquires a personalprofile of each occupant. The personal profile includes, for example,occupant's personal information, which is stored for each occupant. Thepersonal information includes, for example, feature information derivedfrom an image or sounds, a name, an address, an age, a hobby, apreference, past conversation history, and the like. A specific exampleof the personal profile will be described below.

The integrated profile generator 154 generates an integrated profile bycombining at least a part of the personal profiles of the occupants ofthe vehicle M, which have been acquired by the personal profile acquirer152. The integrated profile is an example of “integrated personalinformation”. The integrated profile generator 154 stores the generatedintegrated profile in the storage 170. Details of the functions of thepersonal profile acquirer 152 and the integrated profile generator 154will be described below. The aforementioned personal profile acquirer152 and integrated profile generator 154 may be provided in the manager110.

The pairing application executor 160 performs pairing with the generalpurpose communication device 70 by, for example, Bluetooth (registeredtrademark), and connects the agent function 150-3 and the generalpurpose communication device 70. The agent function 150-3 may beconnected to the general purpose communication device 70 by wiredcommunication using a universal serial bus (USB) and the like.

[Agent Server]

FIG. 5 is a diagram illustrating the configuration of the agent server200 according to the first embodiment and a part of the configuration ofthe agent device 100. Hereinafter, operations of the agent function 150and the like will be described together with the configuration of theagent server 200. Here, a description for physical communication fromthe agent device 100 to the network NW will be omitted. Hereinafter, adescription will be given assuming that the agent function 150-1 isactivated by the WU judger 114 for each agent and a response toutterances is provided to the occupants by the agent 1 implemented bycooperation of the agent function 150-1 and the agent server 200-1;however, for a pair of another agent function and another agent server,there is a difference in functions, database and the like executable ineach of them, but processing is implemented in substantially the sameflow.

The agent server 200-1 includes a communicator 210. The communicator 210is, for example, a network interface such as a network interface card(NIC). Moreover, the agent server 200-1 includes, for example, apersonal profile provider 212, a sound recognizer 220, a naturallanguage processor 222, a conversation manager 224, a network searcher226, a response sentence generator 228, and a storage 250. Thesecomponents are implemented by, for example, a hardware processor such asa CPU that executes a program (software). Some or all of thesecomponents may be implemented by hardware (a circuit unit: includingcircuitry) such as an LSI, an ASIC, a FPGA, and a GPU, or may beimplemented by cooperation of software and hardware. The program may bestored in advance in a storage device (storage device including anon-transitory storage medium) such as an HDD and a flash memory, or maybe installed in the HDD and the flash memory when a detachable storagemedium (non-transitory storage medium) storing the program, such as aDVD and a CD-ROM, is mounted on a drive device. A combination of thesound recognizer 220 and the natural language processor 222 is anexample of a “recognizer”.

The storage 250 is implemented by each of the aforementioned storagedevices. The storage 250 stores, for example, data and programs such asa dictionary DB 252, a personal profile 254, a knowledge base DB 256,and a response rule DB 258.

The personal profile provider 212 acquires occupant's personal profilesfrom the personal profile 254 based on the occupant recognition resulttransmitted by the agent function 150-1, and transmits the acquiredoccupant's personal profiles to the agent function 150-1. The integratedprofile generator 154-1 acquires the occupant's personal profilestransmitted by the agent server 200-1, generates the integrated profile172 from the acquired occupant's personal profiles, and stores thegenerated integrated profile 172 in the storage 170.

The agent function 150-1 transmits, for example, the sound stream inputfrom the acoustic processor 112 and the like or a sound stream subjectedto processing such as compression and encoding to the agent server200-1. When it is possible to recognize a command (request content) thatcan be subjected to local processing (processing not via the agentserver 200-1), the agent function 150-1 may perform a process requestedby the command. The command that can be subjected to local processingis, for example, a command that can be responded to by referring to thestorage 170 included in the agent device 100. More specifically, thecommand that can be subjected to local processing is, for example, acommand that searches for a specific person's name from telephonedirectory data (not illustrated) existing in the storage 170 and calls atelephone number (calls a partner) correlated with a matching nameConsequently, the agent function 150-1 may have some of the functions ofthe agent server 200-1.

When the sound stream is acquired, the sound recognizer 220 performssound recognition and outputs character information converted to a text,and the natural language processor 222 performs semantic interpretationon the character information while referring to the dictionary DB 252.The dictionary DB 252 is, for example, a DB in which abstracted semanticinformation is correlated with the character information. The dictionaryDB 252 includes, for example, a function dictionary 252A and a generalpurpose dictionary 252B. The function dictionary 252A is a dictionaryfor covering functions provided by the agent 1 implemented bycooperation of the agent server 200-1 and the agent function 150-1. Forexample, when the agent 1 provides a function of controlling anin-vehicle air conditioner, words such as “air conditioner”, “airconditioning”, “turn on”, “turn off”, “temperature”, “increase”,“decrease”, “inside air”, and “outside air” are registered in thefunction dictionary 252A in correlation with word types such as verbsand objects, and abstracted meanings. The function dictionary 252A mayinclude inter-word link information indicating that it can besimultaneously used. The general purpose dictionary 252B is a dictionaryin which not only the functions provided by the agent 1 but also eventsof general things are correlated with abstracted meanings. Each of thefunction dictionary 252A and the general purpose dictionary 252B mayinclude list information of synonyms. The function dictionary 252A andthe general purpose dictionary 252B may be prepared corresponding toeach of a plurality of languages, and in such a case, the soundrecognizer 220 and the natural language processor 222 use the functiondictionary 252A and the general purpose dictionary 252B according topreset language setting, and grammar information (not illustrated). Theprocessing of the sound recognizer 220 and the processing of the naturallanguage processor 222 are not clearly divided into steps, and may beperformed while affecting each other such as correction of a recognitionresult by the sound recognizer 220 after the reception of the processingresult of the natural language processor 222.

The natural language processor 222 performs semantic analysis based onthe recognition result of the sound recognizer 220. The natural languageprocessor 222 recognizes meanings such as “What is the weather today?”and “What is the weather?” as the recognition result, and generates acommand replaced with standard character information “Today's weather”.In this way, even when there is a variation in characters in a requestsound, it is possible to easily perform a requested conversation. Forexample, the natural language processor 222 may recognize the meaning ofthe character information by using artificial intelligent processingsuch as machine learning processing using probability, or generate acommand based on a recognition result.

The conversation manager 224 refers to the knowledge base DB 256 and theresponse rule DB 258 based on an input command, determines the contentof a response to the occupants of the vehicle M (for example, thecontent of utterance to the occupants and an image and sounds outputfrom the output). The knowledge base DB 256 is information that definesthe relation between things. The response rule DB 258 is informationthat defines an operation (a response, details of device control, andthe like) to be performed by the agent with respect to the command.

The conversation manager 224 allows the network searcher 226 to performsearching when the command requests information searchable via thenetwork NW. The network searcher 226 accesses various web servers 300via the network NW and acquires desired information. The “informationsearchable via the network NW” is, for example, evaluation results ofgeneral users for a restaurant around the vehicle M or weatherinformation according to the position of the vehicle M on that day.

The response sentence generator 228 generates a response sentence suchthat the content of the utterance determined by the conversation manager224 is transmitted to the occupants of the vehicle M, and transmits thegenerated response sentence (response content) to the agent device 100.The response sentence generator 228 acquires the recognition result ofthe occupant recognition device 80 from the agent device 100, and whenthe acquired recognition result identifies that an occupant who made theutterance including the command is an occupant registered in thepersonal profile 254, the response sentence generator 228 may call thename of the occupant or generate a response sentence in a manner ofspeaking similar to that of the occupant.

When the response sentence is acquired, the agent function 150 instructsthe sound controller 124 to perform sound synthesis and output sounds.The agent function 150 generates an agent image in accordance with thesound output, and instructs the display controller 122 to display thegenerated agent image, an image included in the response content, andthe like. When the response content is displayed as sounds or images,the agent function 150 performs output suitable for the profile of eachoccupant based on the integrated profile 172 stored in the storage 170.By so doing, the agent function in which the virtually appearing agentresponds to the occupants of the vehicle M is implemented.

[Generation of Integrated Profile]

Next, the generation process of the integrated profile 172 describedabove will be specifically described. The personal profile acquirer152-1 transmits the occupant recognition result recognized by theoccupant recognition device 80 to the agent server 200-1 and issues arequest to acquire occupant's personal profiles. The personal profileprovider 212 of the agent server 200-1 refers to the feature informationof the personal profile 254 by using the feature information for eachoccupant included in the occupant recognition result, and acquiresprofile information (personal information) of a corresponding occupant.FIG. 6 is a diagram illustrating an example of the content of thepersonal profile 254. In the personal profile 254, profile informationis correlated with user IDs which are identification information foridentifying users of the agent including the occupants. The profileinformation includes, for example, feature information, a name, anaddress, an age, a hobby, a preference, use history and the like. Thefeature information includes, for example, face feature information,sound feature information, and the like. The face feature informationand the sound feature information include, for example, featureinformation recognizable by the occupant recognition device 80. Theprofile information is registered, for example, at the time of initialsetting of the agent, and is appropriately updated based on an occupantchange instruction, an agent use result, and the like.

The personal profile provider 212 refers to the feature information ofthe personal profile 254 based on the feature information included inthe occupant recognition result, and acquires a personal profileincluding a user ID, a hobby, a preference, and use history correlatedwith feature information with the highest degree of similarity. Thepersonal profile provider 212 may acquire information on a name, anaddress, and an age, in addition to the above information. When a userID (occupant ID) is transmitted from the agent function 150-1, thepersonal profile provider 212 may refer to the user ID of the personalprofile 254 and acquire at least a part of a personal profile correlatedwith a matching user ID. The personal profile provider 212 transmits theacquired occupant's personal profile to the agent function 150-1.

The integrated profile generator 154-1 generates the integrated profile172 by combining at least a part of the personal profile of eachoccupant transmitted from the personal profile provider 212, and storesthe generated integrated profile 172 in the storage 170. FIG. 7 is adiagram illustrating an example of the content of the integrated profile172. In the integrated profile 172, for example, integrated profileinformation is correlated with passenger information that is informationof the occupants in the vehicle M. The integrated profile informationincludes, for example, seating position information, an integratedhobby, an integrated preference, and integrated use history. Thepassenger information is information for identifying the occupantsrecognized by the occupant recognition device 80. In the example of FIG.7, user IDs “U1” and “U2” are stored as the passenger information. Thepassenger information may be the information on the user ID included inthe personal profile acquired by the personal profile acquirer 152-1, ora user ID (occupant ID) input by an occupant through the display andoperation device 20 when the occupant gets on the vehicle M. The seatingposition information is, for example, information on a seating positionof each occupant. The information on the seating position is, forexample, information indicating the driver seat DS, the assistant seatAS, or the rear seats BS1 and BS2 in which occupants is sitting. In theexample of FIG. 7, the seating position information stores informationindicating that an occupant of the user ID “U1” (hereinafter, referredto as an occupant U1) is sitting in the driver seat DS and an occupantof the user ID “U2” (hereinafter, referred to as an occupant U2) issitting in the assistant seat AS. The seating position information maybe, for example, position information (for example, vehicle interiorspace coordinates) for each occupant recognized by the occupantrecognition device 80.

The integrated hobby is information obtained by integrating the hobbiesincluded in the personal profile of each occupant acquired by thepersonal profile acquirer 152-1. The integrated preference isinformation obtained by integrating the preferences included in thepersonal profile of each occupant acquired by the personal profileacquirer 152-1. The integrated use history is information obtained byintegrating the use history included in the personal profile of eachoccupant acquired by the personal profile acquirer 152-1.

The integrated profile generator 154-1 sets a weight for each elementwhen storing the integrated hobby, the integrated preference, and theintegrated use history. The weight is, for example, informationindicating the degree of influence on information provided to theoccupants. The degree of influence is, for example, the importance of anelement when the agent analyzes the meaning of the content of utterancesof the occupant and executes a corresponding function or when the agentoutputs response content based on the execution result of the function.In the present embodiment, for example, when the weight increases, afunction that prioritizes the element is executed, or response contentthat prioritizes the element is output. The weight described above isset for each occupant or each function. When a common element exists ina plurality of occupants, for example, the weight is set by summation.

In the example of FIG. 7, in the integrated hobby, a hobby element<drive> corresponding to the occupant U1 and common hobby elements<music appreciation> and <dining out> of the occupants U1 and U2 arestored. Weights w1 and w2 illustrated in FIG. 7 represent weights foreach element of the hobbies of the occupants U1 and U2. That is,<drive*w1> illustrated in FIG. 7 indicates that the weight w1 has beenset to the hobby element of drive. The w1+w2 illustrated in FIG. 7indicates that the weights w1 and w2 have been added.

<Setting of Weight>

Next, the setting of weights by the integrated profile generator 154-1will be described. For example, when an agent image correlated with anactivated agent is displayed on the display, the integrated profilegenerator 154-1 sets weights based on a displayed area.

FIG. 8 is a diagram for explaining weights set by the integrated profilegenerator 154-1. In the example of FIG. 8, it is assumed that theoccupant U1 is sitting in the drive seat DS of the vehicle M and theoccupant U2 is sitting in the assistant seat AS. FIG. 8 illustrates thefirst display 22 as an example of the display; however, other displaysmay be used. The same applies to the following description.

In the example of FIG. 8, an agent image EI1 correlated with theactivated agent 1 is displayed in the right area when viewed from thefront of the first display 22. In such a case, the integrated profilegenerator 154-1 estimates that the agent image EI1 displayed on thefirst display 22 is displayed at a position visually recognizable fromthe occupant U1 and the occupant U2, based on the positions of theoccupants U1 and U2 recognized by the occupant recognition device 80 andthe display position of the agent image EI1. When the faces of theoccupants U1 and U2 are directed to the display position of the agentimage EI1, the integrated profile generator 154-1 may estimate that theagent image EI1 is displayed at the position visually recognizable fromthe occupant U1 and the occupant U2.

Then, the integrated profile generator 154-1 calculates respectivedistances D1 and D2 from position information of the occupants U1 and U2(positions of the vehicle interior space coordinates) to the displayposition of the agent image EI1 displayed on the first display 22. Then,the integrated profile generator 154-1 sets the weight w1 to be assignedto each element of the profile of the occupant U1 and the weight w2 tobe assigned to each element of the profile of the occupant U2, based onthe calculated respective distances D1 and D2.

In the example of FIG. 8, the distance D1 is shorter than the distanceD2. Consequently, the occupant U1 is closer to the display position ofthe agent image EI1 than the occupant U2. In such a case, the integratedprofile generator 154-1 sets the weight w1 to be larger than the weightw2. In the example of FIG. 8, with respect to the integrated hobby, theintegrated preference, and the integrated use history included in theintegrated profile 172, 0.8 is set for the weight w1 of the element ofthe profile information of the occupant U1 and 0.5 is set for the weightw2 of the element of the profile information of the occupant U2. Thesevalues may be set based on the magnitudes of the distances D1 and D2 ora difference between the distances D1 and D2.

Here, when the display area of the agent image EI1 is moved to the leftside from the display position illustrated in FIG. 8 when viewed fromthe front due to an occupant operation, speaker recognition, and thelike, the agent image EI1 moves away from the occupant U1 and approachesthe occupant U2. Consequently, the integrated profile generator 154-1updates the weights w1 and w2 based on the distances D1 and D2 at thetime point at which the display area of the agent image EI1 has beenchanged.

The integrated profile generator 154-1 may set the weights based on theseating positions of the occupants, instead of (or in addition to) thesetting of the weights based on the display position of the agentdescribed above. For example, the integrated profile generator 154-1sets the weight w1 of each element of the profile of the occupant U1sitting in the drive seat DS to be larger than the weight w2 of eachelement of the profile of the occupant U2 to be seated in the assistantseat AS. When occupants are sitting in the rear seats BS1 and BS2, theintegrated profile generator 154-1 sets the weight of the element of theprofile information of the occupant U1 to be seated in the drive seat DSto be larger than the weight of each element of the profiles of theoccupants seated in the rear seats. This allows the agent 1 to execute afunction and output a response result that prioritizes the profile ofthe occupant U1 to be seated in the drive seat DS. It is predicted thata driver will often make inquiries according to a travel route and atravel state with respect to shops around the current location, ascompared with other occupants. Therefore, by increasing a weight for thedriver, more appropriate information can be provided to the driver.

The integrated profile generator 154-1 may set the predetermined weightof an element of profile information of an occupant to be larger thanthose of other occupants. In this way, irrespective of which of theplurality of occupants speaks the content, a response result specializedfor a specific occupant can be output.

The integrated profile generator 154-1 may change the proportion ofprofile information for each occupant included in the integratedpersonal information, instead of (or in addition to) changing the weightof each element of the integrated profile 172, based on the positions ofthe plurality of occupants and the display position of the agent image.For example, when the distance D1 between the occupant U1 and thedisplay position of the agent image EI1 is shorter than the distance D2between the occupant U2 and the display position of the agent image EI1,the integrated profile 172 is generated by combining the number ofelements of the profile information of the occupant U1 such that thenumber of elements of the profile information of the occupant U1 islarger than the number of elements of the profile information of theoccupant U2.

The generation of the integrated profile 172 described above isperformed by the integrated profile generator 154 of the agent function150 correlated with an activated agent whenever an agent to be activateis switched. That is, the integrated profile 172 generated by the agent1 is controlled such that it is not available in the agent 2 or theagent 3. This allows each agent to output a unique response result. Theintegrated profile 172 may be used by another agent for a fee. In such acase, charging is performed based on the number of uses of theintegrated profile 172 and the amount of data. The charging process maybe performed by the manager 110, for example.

<Information Provision Based on Integrated Profile>

Next, a manner of providing information based on the integrated profile172 will be specifically described. Hereinafter, it is assumed that asan example of information to be provided to a plurality of occupants,information of a response result to the utterance of the occupants isused. FIG. 9 is a diagram for explaining response content when the agentimage EI1 is displayed at a position near the occupant U1. In theexample of FIG. 9, it is assumed that the agent 1 is activated and theoccupant U1 and the occupant U2 are estimated to recognize the agentimage EI1. In such a case, a larger value is set to the weight w1 of theelement of the profile information of the occupant U1 included in theintegrated profile 172 than the weight w2 of the element of the profileinformation of the occupant U2.

Here, when the utterance “Is there recommended restaurant around here?”is received from the occupant U1, the agent 1 (the agent function 150-1and the agent server 200-1) analyzes the meaning of the sound of theutterance and searches for a nearby restaurant. In such a case, theagent 1 may narrow down the search by using some or all of the elementsincluded in the integrated profile 172. In the example of FIG. 7, sinceChinese food and Italian food are included as the elements of theintegrated preference, the agent 1 adds the Chinese food and the Italianfood to keywords and searches for a nearby restaurant. Next, the agent 1generates a response sentence (response content) based on the searchresult, a weight of the integrated profile, and the like, and providesthe generated response sentence to the occupants U1 and U2.

For example, as the weight of an element increases, the agent function150-1 increases the number of response content to be outputcorresponding to the element or emphasizes the response content morethan response content corresponding to another element and allows theoutput controller 120 to output the emphasized response content, andmakes an output sequence faster. In relation to the emphasis and output,for example, in the case of display output, the response content isdisplayed in an emphasized color, is displayed in large characters, oris displayed with a conspicuous gradation, and in the case of soundoutput, the volume of the sound of the response content is set to behigher than another sound.

For example, the agent function 150-1 allows four Chinese restaurants,which are the elements of the profile information of the occupant U1, tobe output and allows one Italian restaurant, which is the element of theprofile information of the occupant U2, to be output, among restaurantresults. In the example of FIG. 9, the agent function 150-1 allows thesound “There are four Chinese restaurants and one Italian restaurantrecommended around here” to be output with a sound image localized atthe display position of the agent image EI1, and allows the firstdisplay 22 to display information on the Chinese and Italian restaurants(for example, restaurant names) together with the agent image EI1. Acharacter image for the Chinese restaurant is displayed to be largerthan a character image for the Italian restaurant.

FIG. 10 is a diagram for explaining response content when the agentimage EI1 is displayed at a position near the occupant U2. In such acase, a larger value is set to the weight w2 of the element of theprofile information of the occupant U2 included in the integratedprofile 172 than the weight w1 of the element of the profile informationof the occupant U1.

Consequently, the agent function 150-1 changes the content of a responseresult based on the weight, allows four Italian restaurants, which arethe elements of the profile information of the occupant U2, to be outputand allows one Chinese restaurant, which is the element of the profileinformation of the occupant U1, to be output, among restaurant results.In the example of FIG. 10, the agent function 150-1 allows the sound“There are four Italian restaurants and one Chinese restaurantrecommended around here” to be output with a sound image localized atthe display position of the agent image EI1, and allows the firstdisplay 22 to display information on the Italian and Chinese restaurantstogether with the agent image EI1. A character image for the Italianrestaurant is displayed to be larger than a character image for theChinese restaurant. As illustrated in FIG. 9 and FIG. 10 describedabove, by providing information using the integrated profile 172, it ispossible to change the response result even for the same utterancecontent, and it is possible to provide more appropriate information tothe occupants. The agent function 150-1, for example, can provide moreappropriate information in providing various types of information suchas reproducing occupant's favorite musical piece or searching for arecommended drive route, in addition to the restaurant search describedabove.

[Processing Flow]

Next, a process performed by the agent system 1 of the first embodimentwill be described. The process performed by the agent system 1 isroughly classified into a process for generating an integrated profileby the agent device 100 (integrated profile generation process) and aprocess for providing information by using the integrated profile(information providing process). Consequently, the respective processeswill be described separately below.

FIG. 11 is a flowchart illustrating an example of the flow of theintegrated profile generation process performed by the agent device 100of the first embodiment. In the example of FIG. 11, it is assumed thatthe agent 1 has already been activated by the utterance of the wake-upword by an occupant and the agent image EI1 has been displayed in apredetermined area of the display. The process of the present flowchartmay be repeatedly performed at a predetermined cycle or at apredetermined timing, for example. The predetermined timing is, forexample, a timing at which the agent 1 has been activated, a timing atwhich the display position of the agent 1 has moved, a timing at whichthe recognition result of the occupant recognition device 80 haschanged, or a timing at which an instruction to generate or update theintegrated profile has been issued by an occupant.

First, the agent device 100 acquires the occupant recognition result ofthe occupant recognition device 80 (step S100). Next, the agent function150-1 acquires the display position of the agent image EI1 displayed bythe display controller 122 (step S102). Next, the integrated profilegenerator 154-1 determines whether an integrated profile has beengenerated (step S104). When it is determined that the integrated profilehas been generated, the personal profile acquirer 152-1 acquires theintegrated profile 172 from the storage 170 (step S106). When it isdetermined that the integrated profile has not been generated, thepersonal profile acquirer 152-1 transmits the occupant recognitionresult to the agent server 200-1 and acquires personal profiles ofoccupants in the vehicle M (step S108). Next, the integrated profilegenerator 154-1 generates an integrated profile by combining at least apart of the acquired personal profiles of the occupants (step S110).

After the process of step S106 or step S110, the integrated profilegenerator 154-1 updates the weight of each element of the integratedprofile based on the display position of the agent image and theposition of each occupant (step S112). Next, the integrated profilegenerator 154-1 stores the updated integrated profile 172 in the storage170 (step S114). In this way, the process of the present flowchart ends.

FIG. 12 is a sequence diagram illustrating an example of the flow of theinformation providing process performed by the agent system 1 of thefirst embodiment. In the example illustrated in FIG. 12, it is assumedthat the information providing process performed by the activated agent1 (the agent device 100 and the agent server 200-1) will be described.In the example illustrated in FIG. 12, it is assumed that the generationor update of the integrated profile 172 has already been performed bythe generation process illustrated in FIG. 11.

First, the acoustic processor 112 of the agent device 100 receives theinput of occupant's utterance from the microphone 10 (step S200) andperforms an acoustic process on the sound of the received utterance(step S202). Next, the agent function 150-1 acquires the integratedprofile 172 stored in the storage 170 (step S204) and transmits thesound subjected to the acoustic process to the agent server 200-1 (stepS206). In the process of step S206, the agent function 150-1 maytransmit the integrated profile.

The sound recognizer 220 of the agent server 200-1 recognizes the sound(sound stream) subjected to the acoustic process transmitted from theagent function 150-1 and converts the sound to a text (step S208). Next,the natural language processor 222 performs a natural language processon the character information converted to the text and performs semanticanalysis of the character information (step S210).

Next, the conversation manager 224 and the network searcher 226 performthe function of the agent based on the semantic analysis result (stepS212). In the process of step S212, the conversation manager 224 and thenetwork searcher 226 may perform a function based on the integratedprofile transmitted from the agent function 150-1. Next, the responsesentence generator 228 generates a response sentence based on theexecuted result (step S214) and transmits the generated responsesentence to the agent device 100 (step S216).

The agent function 150-1 of the agent device 100 selects the responseresult transmitted from the agent server 200-1, based on the integratedprofile and allows the output to output the response result (step S218).

The agent device (information providing device) 100 of the firstembodiment described above includes the personal profile acquirer 152that acquires personal profiles of a plurality of occupants in thevehicle M (an example of a moving body), a processor (the integratedprofile generator 154) that generates an integrated profile by combiningat least a part of the personal profiles of the plurality of occupantsacquired by the personal profile acquirer 152, and an informationprovider (the agent function 150 and the output controller 120) thatprovides information to the plurality of occupants based on theintegrated personal information processed by the processor, therebyproviding more appropriate information to the occupants. Specifically,according to the first embodiment, when the plurality of occupants arein the vehicle M, it is possible to output a response result thatprioritizes the display position of the agent and the profileinformation of a conversation target person.

Second Embodiment

Hereinafter, the second embodiment will be described. The agent system 1of the second embodiment is different from the first embodiment in thatthe integrated profile generation process performed by the agent device100 is performed by an agent server side and a generated integratedprofile is stored in the storage 250A of the agent server 200A.Consequently, the difference described above will be mainly describedbelow. The same components as those in the first embodiment are denotedby the same names or reference numerals and a detailed descriptionthereof will be omitted. In the second embodiment, the agent server isan example of an information providing device.

[Agent Device]

FIG. 13 is a diagram illustrating a configuration of an agent device100A according to the second embodiment and devices mounted on thevehicle M. In the vehicle M, for example, one or more microphones 10, adisplay and operation device 20, a speaker unit 30, a navigation device40, a vehicle device 50, an in-vehicle communication device 60, anoccupant recognition device 80, and the agent device 100A are mounted.There is a case where a general purpose communication device 70 isbrought into a vehicle interior and is used as a communication device.

The agent device 100A includes a manager 110, agent functions 150A-1 to150A-3, a pairing application executor 160, and a storage 170. The agentfunctions 150A-1 to 150A-3 are different from the agent functions 150-1to 150-3 of the first embodiment described above in terms of functionsin that they do not include the configurations of the personal profileacquirers 152-1 to 152-3 and the integrated profile generators 154-1 to154-3, respectively. Consequently, each of the agent functions 150A-1 to150A-3 performs processes other than the processes performed by thepersonal profile acquirers 152-1 to 152-3 and the integrated profilegenerators 154-1 to 154-3. The agent functions 150A-1 to 150A-3 transmitinformation on the occupant recognition result of the occupantrecognition device 80 and the display position of the agent image to theagent server 200A-1 at a predetermined cycle or at a predeterminedtiming.

[Agent Server]

FIG. 14 is a diagram illustrating the configuration of the agent server200A-1 according to the second embodiment and a part of theconfiguration of the agent device 100A. Similarly to the firstembodiment, also in the second embodiment, the agent function 150A-1 isactivated and a response to utterance is provided to an occupant by theagent 1 implemented by cooperation of the agent function 150A-1 and theagent server 200A-1.

The agent server 200A-1 includes a communicator 210, an integratedprofile generator 214, a sound recognizer 220, a natural languageprocessor 222, a conversation manager 224, a network searcher 226, aresponse sentence generator 228, and the storage 250A. These componentsare implemented by, for example, a hardware processor such as a CPU thatexecutes a program (software). Some or all of these components may beimplemented by hardware (a circuit unit: including circuitry) such as anLSI, an ASIC, a FPGA, and a GPU, or may be implemented by cooperation ofsoftware and hardware. The program may be stored in advance in a storagedevice (storage device including a non-transitory storage medium) suchas an HDD and a flash memory, or may be installed in the HDD and theflash memory when a detachable storage medium (non-transitory storagemedium) storing the program, such as a DVD and a CD-ROM, is mounted on adrive device. In the second embodiment, a combination of theconversation manager 224, the network searcher 226, and the responsesentence generator 228 is an example of an “information provider”.

The storage 250A is implemented by the aforementioned each storagedevice. The storage 250A stores, for example, data and programs such asa dictionary DB 252, a personal profile 254, a knowledge base DB 256, aresponse rule DB 258, and an integrated profile 260. The integratedprofile 260 stores, for example, information on items similar to thoseof the integrated profile 172 of the first embodiment.

The integrated profile generator 214 refers to the personal profile 254based on feature information for each occupant included in the occupantrecognition result transmitted by the agent function 150A-1, andacquires a personal profile of each occupant. Then, the integratedprofile generator 214 generates an integrated profile by combining atleast a part of the acquired personal profile of each occupant.Moreover, the integrated profile generator 214 sets a weight (degree ofinfluence) for each element, which is included in the generatedintegrated profile, based on the positions of the occupants and adisplay position of an agent image. Then, the integrated profilegenerator 214 stores the integrated profile 260 with the set weights inthe storage 250A. For example, when there is a change in the occupantrecognition result transmitted by the agent function 150A-1 or thedisplay position of the agent image, the integrated profile generator214 updates the integrated profile.

The information provider of the agent server 200A-1 acquires theintegrated profile 260 stored in the storage 250A when performingconversation management, network search, and response sentencegeneration based on the semantic analysis result of sounds processed bythe natural language processor 222, and changes functions to beperformed by an agent and response content (response sentence) based onthe acquired integrated profile 260. The information provider transmitsthe generated response content to the agent device 100A and providesinformation based on the response content to the occupants.

According to the agent system of the second embodiment described above,the agent server 200A-1 performs the generation and update of theintegrated profile and the generation of response content based on theintegrated profile, so that it is possible to reduce a processing loadon the agent device 100A side, in addition to the same effects as thosein the agent system of the first embodiment.

Each of the first embodiment and the second embodiment described abovemay combine some or all of other embodiments. Some or all of thefunctions of the agent device 100 (100A) may be included in the agentserver 200 (200A-1). Some or all of the functions of the agent server200 (200A-1) may be included in the agent device 100 (100A). That is,the separation of the functions in the agent device 100 (100A) and theagent server 200 (200A-1) may be appropriately changed by the componentsof each device, the scale of the agent server 200 (200A-1) and the agentsystem 1, and the like. The separation of the functions in the agentdevice 100 (100A) and the agent server 200 (200A-1) may be set for eachvehicle M or each agent mounted on the vehicle.

In the embodiments described above, a vehicle is used as an example of amoving body; however, another moving body such as a ship and a flyingobject may be used.

Although a mode for carrying out the present invention have beendescribed using the embodiments, the present invention is not limited tosuch embodiments and various modifications and replacements can be madewithout departing from the scope of the present invention.

What is claimed is:
 1. An information providing device comprising: anacquirer configured to acquire personal information of a plurality ofoccupants in a moving body; a processor configured to generateintegrated personal information by combining at least a part of thepersonal information of the plurality of occupants acquired by theacquirer; and an information provider configured to provide informationto the plurality of occupants based on the integrated personalinformation processed by the processor; wherein the information providerfurther comprise: a display controller configured to control a displayto display an agent image that communicates with the plurality ofoccupants; and an agent functional unit configured to provide a serviceincluding allowing an output to output a sound response according tosounds of the plurality of occupants, wherein the processor changes adegree of influence of the personal information of the occupantsincluded in the integrated personal information based on a displayposition of the agent image and respective positions of the plurality ofoccupants in the moving body.
 2. The information providing deviceaccording to claim 1, wherein the processor combines at least a part ofthe personal information of the plurality of occupants based on an areaof the display where the agent image is displayed.
 3. The informationproviding device according to claim 1, wherein the processor increasesthe degree of influence of the personal information of the occupantsincluded in the integrated personal information with respect to anoccupant located at a position near the display position of the agentimage among the plurality of occupants.
 4. The information providingdevice according to claim 1, wherein the processor increases the degreeof influence of the personal information of the occupants included inthe integrated personal information with respect to an occupant locatedat a position near the display position of the agent image among theplurality of occupants.
 5. The information providing device according toclaim 1, wherein the processor changes a ratio of the personalinformation of the occupants included in the integrated personalinformation based on the display position of the agent image andrespective positions of the plurality of occupants in the moving body.6. The information providing device according to claim 2, wherein thedisplay has a display area large enough for two or more of the pluralityof occupants to recognize that the agent image is displayed at aposition facing the two or more occupants.
 7. An information providingmethod comprising: acquiring, by an information providing device,personal information of a plurality of occupants in a moving body;generating, by the information providing device, integrated personalinformation by combining at least a part of the acquired personalinformation of the plurality of occupants; providing, by the informationproviding device, information to the plurality of occupants based on thegenerated integrated personal information; controlling, by theinformation providing device, a display to display an agent image thatcommunicates with the plurality of occupants; providing, by theinformation providing device, a service including allowing an output tooutput a sound response according to sounds of the plurality ofoccupants; and changing, by the information providing device, a degreeof influence of the personal information of the occupants included inthe integrated personal information based on a display position of theagent image and respective positions of the plurality of occupants inthe moving body.
 8. An information providing device comprising: anacquirer configured to acquire personal information of a plurality ofoccupants in a moving body; a processor configured to generateintegrated personal information by combining at least a part of thepersonal information of the plurality of occupants acquired by theacquirer; and an information provider configured to provide informationto the plurality of occupants based on the integrated personalinformation processed by the processor; wherein the information providerfurther comprises: a display controller configured to control a displayto display an agent image that communicates with the plurality ofoccupants; and an agent functional unit configured to provide a serviceincluding allowing an output to output a sound response according tosounds of the plurality of occupants, wherein the display has a displayarea large enough for two or more of the plurality of occupants torecognize that the agent image is displayed at a position facing the twoor more occupants.
 9. An information providing method comprising:acquiring, by an information providing device, personal information of aplurality of occupants in a moving body; generating, by the informationproviding device, integrated personal information by combining at leasta part of the personal information of the plurality of occupants;providing, by the information providing device, information to theplurality of occupants based on the integrated personal information;controlling, by the information providing device, a display to displayan agent image that communicates with the plurality of occupants; andproviding, by the information providing device, a service includingallowing an output to output a sound response according to sounds of theplurality of occupants, wherein the display has a display area largeenough for two or more of the plurality of occupants to recognize thatthe agent image is displayed at a position facing the two or moreoccupants.